Semantic Grouping Network for Video Captioning

This paper considers a video caption generating network referred to as Semantic Grouping Network (SGN) that attempts (1) group frames with discriminating word phrases of partially decoded and then (2) decode those semantically aligned groups in predicting the next word. As consecutive are not likely provide unique information, prior methods have focused on discarding or merging repetitive information based only input video. The SGN learns an algorithm capture most mapping associates each phrase relevant - establishing this allows related be clustered, which reduces redundancy. In contrast methods, continuous feedback from words enables dynamically update representation adapts caption. Furthermore, contrastive attention loss is proposed facilitate accurate alignment between without manual annotations. achieves state-of-the-art performances by outperforming runner-up margin 2.1%p 2.4%p CIDEr-D score MSVD MSR-VTT datasets, respectively. Extensive experiments demonstrate effectiveness interpretability SGN.